Report batches that were never sealed instead of completing them - #13

Draft — against my own fork for discussion, not upstream.

Maintenance could complete a batch whose transaction was still open or rolled back — firing on_success for work that never happened and losing the deferred enqueues to AlreadyFinished. This PR makes that impossible, without giving up crash recovery.

The three scenarios, as integration tests in batch_lifecycle_test.rb (each runs a real dispatcher and worker):

1. Transaction outlives the stalled window → now works

JobResult.transaction do
  JobResult.create!(...)
  batch = SolidQueue::Batch.enqueue(on_success: BatchOnSuccessJob.new("late")) do
    AddToBufferJob.perform_later("late")            # deferred until commit
  end

  SolidQueue::Batch.where(id: batch.id).update_all(created_at: 10.minutes.ago)
  SolidQueue::Batch.sweep_stalled                   # maintenance on another process

  assert_not batch.reload.finished?                 # ← was: finished, on_success fired
end

wait_for_batches_to_finish_for(5.seconds)
assert_equal [ "late", "late: 1 jobs succeeded!" ], JobBuffer.values.sort   # ← was: AlreadyFinished at commit

2. Rollback → no phantom success

JobResult.transaction do
  batch = SolidQueue::Batch.enqueue(on_success: BatchOnSuccessJob.new("phantom")) do
    AddToBufferJob.perform_later("rolled back")
  end
  raise ActiveRecord::Rollback
end

SolidQueue::Batch.where(id: batch.id).update_all(created_at: 10.minutes.ago)
SolidQueue::Batch.sweep_stalled

assert_not batch.reload.finished?                   # reported now, removed after a day
assert_not_includes JobBuffer.values, "phantom: 1 jobs succeeded!"   # ← was: fired

3. Server dies right after enqueue → nothing to repair

SolidQueue::Batch.any_instance.stubs(:start)        # the deferred start never runs

batch = SolidQueue::Batch.enqueue(on_success: BatchOnSuccessJob.new("sealed")) do
  AddToBufferJob.perform_later("sealed")
end

assert batch.reload.enqueued?                       # sealed in the same commit as its row

wait_for_batches_to_finish_for(5.seconds)
assert_equal [ "sealed", "sealed: 1 jobs succeeded!" ], JobBuffer.values.sort

How

Two changes:

Maintenance completes only batches their creator sealed (called start on). Unsealed ones follow a lifecycle instead of a guess:

• 5 minutes: reported — a stalled_batch.solid_queue event each, visible via SolidQueue::Batch.stalled, resolvable early with .start or .destroy.
• 1 day (expire_after): removed. A batch still unsealed then came from a transaction that will never commit; its data rolled back, so the batch ends the same way — as if never created. No callbacks, no accumulation, nothing to monitor.

Jobs that reached the queue before a rollback already ran and stay untouched — the same behaviour as non-batch jobs enqueued in a rolled-back transaction. A transaction that outlives the expiry and then commits raises AlreadyFinished from its deferred enqueues: loud, and without success callbacks over lost work.

A batch created outside any transaction seals itself atomically:

seal_on_create = ActiveRecord.all_open_transactions.none?   # before opening our own txn

transaction do
  save! if new_record?
  wrap_in_batch_context(id) { block.call }
  seal_if_filled if seal_on_create    # UPDATE ... SET enqueued_at WHERE total_jobs != 0
end

Crash → either nothing committed or a sealed batch and its jobs committed together, which is why scenario 3 needs no sweeper.

The total_jobs != 0 guard: deferred enqueues put no jobs in the batch's own transaction, and a sealed batch with nothing outstanding means complete — sealing there would let a concurrent sweep finish it before the deferred INSERTs land. So all-deferred configurations (Rails 7.2 :default, 8.2 defaults) keep sealing via the deferred start; a crash in that window goes to the reported path, since the jobs existed only in process memory and there is nothing durable to recover.

Rails 7.1 (no all_open_transactions) is unchanged.

Verification

Suites green on Rails 7.1 / 7.2 / 8.0 / 8.1 / main, PostgreSQL, shared-pool and separate-pool layouts, rubocop clean. Companion: #7 deletes the leftover pending rows from scenario 2.

🤖 Generated with Claude Code